{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<font color=\"red\" size=\"5\"><strong>!Important!</strong> </br>This notebook is outdated and is not supported by the AutoML Team. Please use the supported version ([link](https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/automl-standalone-jobs/automl-forecasting-task-bike-share)).</font>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Automated Machine Learning\n",
        "**BikeShare Demand Forecasting**\n",
        "\n",
        "## Contents\n",
        "1. [Introduction](#Introduction)\n",
        "1. [Setup](#Setup)\n",
        "1. [Compute](#Compute)\n",
        "1. [Data](#Data)\n",
        "1. [Train](#Train)\n",
        "1. [Featurization](#Featurization)\n",
        "1. [Evaluate](#Evaluate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Introduction\n",
        "This notebook demonstrates demand forecasting for a bike-sharing service using AutoML.\n",
        "\n",
        "AutoML highlights here include built-in holiday featurization, accessing engineered feature names, and working with the `forecast` function. Please also look at the additional forecasting notebooks, which document lagging, rolling windows, forecast quantiles, other ways to use the forecast function, and forecaster deployment.\n",
        "\n",
        "Make sure you have executed the [configuration notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) before running this notebook.\n",
        "\n",
        "Notebook synopsis:\n",
        "1. Creating an Experiment in an existing Workspace\n",
        "2. Configuration and local run of AutoML for a time-series model with lag and holiday features \n",
        "3. Viewing the engineered names for featurized data and featurization summary for all raw features\n",
        "4. Evaluating the fitted model using a rolling test "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1680248038565
        }
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import logging\n",
        "from datetime import datetime\n",
        "\n",
        "import azureml.core\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from azureml.automl.core.featurization import FeaturizationConfig\n",
        "from azureml.core import Dataset, Experiment, Workspace\n",
        "from azureml.train.automl import AutoMLConfig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This notebook is compatible with Azure ML SDK version 1.35.0 or later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "ws = Workspace.from_config()\n",
        "\n",
        "# choose a name for the run history container in the workspace\n",
        "experiment_name = \"automl-bikeshareforecasting\"\n",
        "\n",
        "experiment = Experiment(ws, experiment_name)\n",
        "\n",
        "output = {}\n",
        "output[\"Subscription ID\"] = ws.subscription_id\n",
        "output[\"Workspace\"] = ws.name\n",
        "output[\"SKU\"] = ws.sku\n",
        "output[\"Resource Group\"] = ws.resource_group\n",
        "output[\"Location\"] = ws.location\n",
        "output[\"Run History Name\"] = experiment_name\n",
        "output[\"SDK Version\"] = azureml.core.VERSION\n",
        "pd.set_option(\"display.max_colwidth\", None)\n",
        "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
        "outputDf.T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Compute\n",
        "You will need to create a [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute) for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
        "\n",
        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
        "\n",
        "#### Creation of AmlCompute takes approximately 5 minutes. \n",
        "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "# Choose a name for your cluster.\n",
        "amlcompute_cluster_name = \"bike-cluster\"\n",
        "\n",
        "# Verify that cluster does not exist already\n",
        "try:\n",
        "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
        "    print(\"Found existing cluster, use it.\")\n",
        "except ComputeTargetException:\n",
        "    compute_config = AmlCompute.provisioning_configuration(\n",
        "        vm_size=\"STANDARD_DS12_V2\", max_nodes=4\n",
        "    )\n",
        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
        "\n",
        "compute_target.wait_for_completion(show_output=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Data\n",
        "\n",
        "Let's set up what we know about the dataset. \n",
        "\n",
        "**Target column** is what we want to forecast.\n",
        "\n",
        "**Time column** is the time axis along which to predict."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "target_column_name = \"cnt\"\n",
        "time_column_name = \"date\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "source": [
        "You are now ready to load the historical bike share data. We will load the CSV file into a plain pandas DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "outputs": [],
      "source": [
        "all_data = pd.read_csv(\"bike-no.csv\", parse_dates=[time_column_name])\n",
        "\n",
        "# Drop the columns 'casual' and 'registered' as these columns are a breakdown of the total and therefore a leak.\n",
        "all_data.drop([\"casual\", \"registered\"], axis=1, inplace=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "source": [
        "### Split the data\n",
        "\n",
        "The first split we make is into train and test sets. Note we are splitting on time. Data before 9/1 will be used for training, and data after and including 9/1 will be used for testing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1680247376789
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "outputs": [],
      "source": [
        "# select data that occurs before a specified date\n",
        "train = all_data[all_data[time_column_name] <= pd.Timestamp(\"2012-08-31\")].copy()\n",
        "test = all_data[all_data[time_column_name] >= pd.Timestamp(\"2012-09-01\")].copy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Upload data to datastore\n",
        "\n",
        "The [Machine Learning service workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-workspace) is paired with the storage account, which contains the default data store. We will use it to upload the bike share data and create [tabular dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py) for training. A tabular dataset defines a series of lazily-evaluated, immutable operations to load data from the data source into tabular representation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "outputs": [],
      "source": [
        "from azureml.data.dataset_factory import TabularDatasetFactory\n",
        "\n",
        "datastore = ws.get_default_datastore()\n",
        "\n",
        "train_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
        "    train, target=(datastore, \"dataset/\"), name=\"bike_no_train\"\n",
        ")\n",
        "\n",
        "test_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
        "    test, target=(datastore, \"dataset/\"), name=\"bike_no_test\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Forecasting Parameters\n",
        "To define forecasting parameters for your experiment training, you can leverage the ForecastingParameters class. The table below details the forecasting parameter we will be passing into our experiment.\n",
        "\n",
        "|Property|Description|\n",
        "|-|-|\n",
        "|**time_column_name**|The name of your time column.|\n",
        "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
        "|**country_or_region_for_holidays**|The country/region used to generate holiday features. These should be ISO 3166 two-letter country/region codes (i.e. 'US', 'GB').|\n",
        "|**target_lags**|The target_lags specifies how far back we will construct the lags of the target variable.|\n",
        "|**freq**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information.\n",
        "|**cv_step_size**|Number of periods between two consecutive cross-validation folds. The default value is \"auto\", in which case AutoMl determines the cross-validation step size automatically, if a validation set is not provided. Or users could specify an integer value."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train\n",
        "\n",
        "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
        "\n",
        "|Property|Description|\n",
        "|-|-|\n",
        "|**task**|forecasting|\n",
        "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>\n",
        "|**blocked_models**|Models in blocked_models won't be used by AutoML. All supported models can be found at [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting?view=azure-ml-py).|\n",
        "|**experiment_timeout_hours**|Experimentation timeout in hours.|\n",
        "|**training_data**|Input dataset, containing both features and label column.|\n",
        "|**label_column_name**|The name of the label column.|\n",
        "|**compute_target**|The remote compute for training.|\n",
        "|**n_cross_validations**|Number of cross-validation folds to use for model/pipeline selection. The default value is \"auto\", in which case AutoMl determines the number of cross-validations automatically, if a validation set is not provided. Or users could specify an integer value.\n",
        "|**enable_early_stopping**|If early stopping is on, training will stop when the primary metric is no longer improving.|\n",
        "|**forecasting_parameters**|A class that holds all the forecasting related parameters.|\n",
        "\n",
        "This notebook uses the blocked_models parameter to exclude some models that take a longer time to train on this dataset. You can choose to remove models from the blocked_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Setting forecaster maximum horizon \n",
        "\n",
        "The forecast horizon is the number of periods into the future that the model should predict. Here, we set the horizon to 14 periods (i.e. 14 days). Notice that this is much shorter than the number of days in the test set; we will need to use a rolling test to evaluate the performance on the whole test set. For more discussion of forecast horizons and guiding principles for setting them, please see the [energy demand notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand).  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "forecast_horizon = 14"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Convert prediction type to integer\n",
        "The featurization configuration can be used to change the default prediction type from decimal numbers to integer. This customization can be used in the scenario when the target column is expected to contain whole values as the number of rented bikes per day."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "featurization_config = FeaturizationConfig()\n",
        "# Force the target column, to be integer type.\n",
        "featurization_config.add_prediction_transform_type(\"Integer\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Config AutoML"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
        "\n",
        "forecasting_parameters = ForecastingParameters(\n",
        "    time_column_name=time_column_name,\n",
        "    forecast_horizon=forecast_horizon,\n",
        "    country_or_region_for_holidays=\"US\",  # set country_or_region will trigger holiday featurizer\n",
        "    target_lags=\"auto\",  # use heuristic based lag setting\n",
        "    freq=\"D\",  # Set the forecast frequency to be daily\n",
        "    cv_step_size=\"auto\",\n",
        ")\n",
        "\n",
        "automl_config = AutoMLConfig(\n",
        "    task=\"forecasting\",\n",
        "    primary_metric=\"normalized_root_mean_squared_error\",\n",
        "    featurization=featurization_config,\n",
        "    blocked_models=[\"ExtremeRandomTrees\"],\n",
        "    experiment_timeout_hours=0.3,\n",
        "    training_data=train_dataset,\n",
        "    label_column_name=target_column_name,\n",
        "    compute_target=compute_target,\n",
        "    enable_early_stopping=True,\n",
        "    n_cross_validations=\"auto\",  # Feel free to set to a small integer (>=2) if runtime is an issue.\n",
        "    max_concurrent_iterations=4,\n",
        "    max_cores_per_iteration=-1,\n",
        "    verbosity=logging.INFO,\n",
        "    forecasting_parameters=forecasting_parameters,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will now run the experiment, you can go to Azure ML portal to view the run details. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "remote_run = experiment.submit(automl_config, show_output=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "remote_run.wait_for_completion()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Retrieve the Best Run details\n",
        "Below we retrieve the best Run object from among all the runs in the experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_run = remote_run.get_best_child()\n",
        "best_run"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Featurization\n",
        "\n",
        "We can look at the engineered feature names generated in time-series featurization via. the JSON file named 'engineered_feature_names.json' under the run outputs. Note that a number of named holiday periods are represented. We recommend that you have at least one year of data when using this feature to ensure that all yearly holidays are captured in the training featurization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Download the JSON file locally\n",
        "best_run.download_file(\n",
        "    \"outputs/engineered_feature_names.json\", \"engineered_feature_names.json\"\n",
        ")\n",
        "with open(\"engineered_feature_names.json\", \"r\") as f:\n",
        "    records = json.load(f)\n",
        "\n",
        "records"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### View the featurization summary\n",
        "\n",
        "You can also see what featurization steps were performed on different raw features in the user data. For each raw feature in the user data, the following information is displayed:\n",
        "\n",
        "- Raw feature name\n",
        "- Number of engineered features formed out of this raw feature\n",
        "- Type detected\n",
        "- If feature was dropped\n",
        "- List of feature transformations for the raw feature"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Download the featurization summary JSON file locally\n",
        "best_run.download_file(\n",
        "    \"outputs/featurization_summary.json\", \"featurization_summary.json\"\n",
        ")\n",
        "\n",
        "# Render the JSON as a pandas DataFrame\n",
        "with open(\"featurization_summary.json\", \"r\") as f:\n",
        "    records = json.load(f)\n",
        "fs = pd.DataFrame.from_records(records)\n",
        "\n",
        "# View a summary of the featurization\n",
        "fs[\n",
        "    [\n",
        "        \"RawFeatureName\",\n",
        "        \"TypeDetected\",\n",
        "        \"Dropped\",\n",
        "        \"EngineeredFeatureCount\",\n",
        "        \"Transformations\",\n",
        "    ]\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Evaluate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now use the best fitted model from the AutoML Run to make forecasts for the test set. We will do batch scoring on the test dataset which should have the same schema as training dataset.\n",
        "\n",
        "The scoring will run on a remote compute. In this example, it will reuse the training compute."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "test_experiment = Experiment(ws, experiment_name + \"_test\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Retrieving forecasts from the model\n",
        "To run the forecast on the remote compute we will use a helper script: forecasting_script. This script contains the utility methods which will be used by the remote estimator. We copy the script to the project folder to upload it to remote compute."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "import shutil\n",
        "\n",
        "script_folder = os.path.join(os.getcwd(), \"forecast\")\n",
        "os.makedirs(script_folder, exist_ok=True)\n",
        "shutil.copy(\"forecasting_script.py\", script_folder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For brevity, we have created a function called run_forecast that submits the test data to the best model determined during the training run and retrieves forecasts. The test set is longer than the forecast horizon specified at train time, so the forecasting script uses a so-called rolling evaluation to generate predictions over the whole test set. A rolling evaluation iterates the forecaster over the test set, using the actuals in the test set to make lag features as needed. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from run_forecast import run_rolling_forecast\n",
        "\n",
        "remote_run = run_rolling_forecast(\n",
        "    test_experiment, compute_target, best_run, test_dataset, target_column_name\n",
        ")\n",
        "remote_run"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "remote_run.wait_for_completion(show_output=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Download the prediction result for metrics calculation\n",
        "The test data with predictions are saved in artifact outputs/predictions.csv. You can download it and calculation some error metrics for the forecasts and vizualize the predictions vs. the actuals."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "remote_run.download_file(\"outputs/predictions.csv\", \"predictions.csv\")\n",
        "fcst_df = pd.read_csv(\"predictions.csv\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Note that the rolling forecast can contain multiple predictions for each date, each from a different forecast origin. For example, consider 2012-09-05:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "fcst_df[fcst_df.date == \"2012-09-05\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here, the forecast origin refers to the latest date of actuals available for a given forecast. The earliest origin in the rolling forecast, 2012-08-31, is the last day in the training data. For origin date 2012-09-01, the forecasts use actual recorded counts from the training data *and* the actual count recorded on 2012-09-01. Note that the model is not retrained for origin dates later than 2012-08-31, but the values for model features, such as lagged values of daily count, are updated.\n",
        "\n",
        "Let's calculate the metrics over all rolling forecasts:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.automl.core.shared import constants\n",
        "from azureml.automl.runtime.shared.score import scoring\n",
        "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
        "\n",
        "# use automl metrics module\n",
        "scores = scoring.score_regression(\n",
        "    y_test=fcst_df[target_column_name],\n",
        "    y_pred=fcst_df[\"predicted\"],\n",
        "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET),\n",
        ")\n",
        "\n",
        "print(\"[Test data scores]\\n\")\n",
        "for key, value in scores.items():\n",
        "    print(\"{}:   {:.3f}\".format(key, value))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For more details on what metrics are included and how they are calculated, please refer to [supported metrics](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#regressionforecasting-metrics). You could also calculate residuals, like described [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#residuals).\n",
        "\n",
        "The rolling forecast metric values are very high in comparison to the validation metrics reported by the AutoML job. What's going on here? We will investigate in the following cells!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Forecast versus actuals plot\n",
        "We will plot predictions and actuals on a time series plot. Since there are many forecasts for each date, we select the 14-day-ahead forecast from each forecast origin for our comparison."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from matplotlib import pyplot as plt\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "fcst_df_h14 = (\n",
        "    fcst_df.groupby(\"forecast_origin\", as_index=False)\n",
        "    .last()\n",
        "    .drop(columns=[\"forecast_origin\"])\n",
        ")\n",
        "fcst_df_h14.set_index(time_column_name, inplace=True)\n",
        "plt.plot(fcst_df_h14[[target_column_name, \"predicted\"]])\n",
        "plt.xticks(rotation=45)\n",
        "plt.title(f\"Predicted vs. Actuals\")\n",
        "plt.legend([\"actual\", \"14-day-ahead forecast\"])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Looking at the plot, there are two clear issues:\n",
        "1. An anomalously low count value on October 29th, 2012.\n",
        "2. End-of-year holidays (Thanksgiving and Christmas) in late November and late December.\n",
        "\n",
        "What happened on Oct. 29th, 2012? That day, Hurricane Sandy brought severe storm surge flooding to the east coast of the United States, particularly around New York City. This is certainly an anomalous event that the model did not account for!\n",
        "\n",
        "As for the late year holidays, the model apparently did not learn to account for the full reduction of bike share rentals on these major holidays. The training data covers 2011 and early 2012, so the model fit only had access to a single occurrence of these holidays. This makes it challenging to resolve holiday effects; however, a larger AutoML model search may result in a better model that is more holiday-aware.\n",
        "\n",
        "If we filter the predictions prior to the Thanksgiving holiday and remove the anomalous day of 2012-10-29, the metrics are closer to validation levels:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "date_filter = (fcst_df.date != \"2012-10-29\") & (fcst_df.date < \"2012-11-22\")\n",
        "scores = scoring.score_regression(\n",
        "    y_test=fcst_df[date_filter][target_column_name],\n",
        "    y_pred=fcst_df[date_filter][\"predicted\"],\n",
        "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET),\n",
        ")\n",
        "\n",
        "print(\"[Test data scores (filtered)]\\n\")\n",
        "for key, value in scores.items():\n",
        "    print(\"{}:   {:.3f}\".format(key, value))"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "jialiu"
      }
    ],
    "category": "tutorial",
    "compute": [
      "Remote"
    ],
    "datasets": [
      "BikeShare"
    ],
    "deployment": [
      "None"
    ],
    "exclude_from_index": false,
    "file_extension": ".py",
    "framework": [
      "Azure ML AutoML"
    ],
    "friendly_name": "Forecasting BikeShare Demand",
    "index_order": 1,
    "kernel_info": {
      "name": "python38-azureml"
    },
    "kernelspec": {
      "display_name": "Python 3.8 - AzureML",
      "language": "python",
      "name": "python38-azureml"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "microsoft": {
      "ms_spell_check": {
        "ms_spell_check_language": "en"
      }
    },
    "mimetype": "text/x-python",
    "name": "python",
    "npconvert_exporter": "python",
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    },
    "pygments_lexer": "ipython3",
    "tags": [
      "Forecasting"
    ],
    "task": "Forecasting",
    "version": 3,
    "vscode": {
      "interpreter": {
        "hash": "6bd77c88278e012ef31757c15997a7bea8c943977c43d6909403c00ae11d43ca"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}